Skip to content

Test offline Stim decoding outputs on two NCCL ranks - #102

Open
ivanbasov wants to merge 1 commit into
mainfrom
test/offline-stim-nccl-outputs
Open

ivanbasov wants to merge 1 commit into
mainfrom
test/offline-stim-nccl-outputs

Conversation

@ivanbasov

@ivanbasov ivanbasov commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

#100 gathers the PREDECODER_DECODE_OUTPUT_DIR arrays across ranks with torch.distributed.gather_object, but that path is only tested on the gloo backend (CPU). multi-gpu-tests runs training and inference on generated samples, so it does not reach it either.

This adds TestOfflineStimDecodeNCCL to code/tests/test_multi_gpu.py. It writes Stim sample files for X and Z (257 shots, so the remainder shot is exercised), decodes them on two NCCL ranks in both decode modes, and compares the reported counts and every saved per-shot array with a single-process run on the same files. Helpers are reused from test_offline_stim_decoding.py.

Test-only change, 132 added lines.

Verification

  • Run manually on a node with 4x GB200 (torch 2.12, NCCL 2.29.7) at e382271:

    PYTHONPATH=code python -m unittest discover -s code/tests -p test_multi_gpu.py -v
    ...
    test_counts_and_saved_arrays_match_single_process (test_multi_gpu.TestOfflineStimDecodeNCCL...) ... ok
    Ran 4 tests in 26.815s
    OK
    
  • On a single-GPU host the new class is skipped like the rest of the module (Ran 4 tests, OK (skipped=4)).

  • multi-gpu-tests only runs on main, so on this PR the new test is skipped in CI; it will first run there after merge.

🤖 Generated with Claude Code

The gathered PREDECODER_DECODE_OUTPUT_DIR arrays added in #100 are only
tested on the gloo backend. Add a multi-GPU test that decodes the same
Stim sample files on two NCCL ranks and compares the counts and the saved
per-shot arrays with a single-process run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Ivan Basov <ibasov@nvidia.com>
@ivanbasov
ivanbasov marked this pull request as ready for review November 17, 2026 21:35
@ivanbasov
ivanbasov requested review from bmhowe23 and kvmto November 17, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant